Skip to content

feat: Add an option for specifying model name #685

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: Add an option for specifying model name #685

wants to merge 1 commit into from

Conversation

vrdn-23
Copy link
Contributor

@vrdn-23 vrdn-23 commented Jul 24, 2025

What does this PR do?

This PR introduces a new CLI argument, --served-model-name, which allows users to specify a custom model name to be returned in responses from the OpenAI-compatible endpoint.

This is particularly useful in scenarios where the model is loaded from a local path (e.g., /data/model) and does not have an inherent name associated with it. By setting --served-model-name, users can override the default model identifier (which might be a generic or filesystem-based value) and provide a more descriptive or meaningful name in the API response. This helps improve clarity and consistency, especially when integrating with clients or tools that rely on the model field in the response for tracking or routing purposes.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

@Narsil @alvarobartt @kozistr

I have tested this to the best of my ability, but I'm not sure if I did the gRPC bits correctly, so if someone could help verify that, that would be great!

@vrdn-23 vrdn-23 mentioned this pull request Jul 25, 2025
Merged
5 tasks
@vrdn-23
Copy link
Contributor Author

vrdn-23 commented Aug 6, 2025

@alvarobartt Let me know if you have are any changes/feedback for this PR

@vrdn-23
Copy link
Contributor Author

vrdn-23 commented Aug 19, 2025

@alvarobartt @Narsil I was wondering if you have any thoughts/concerns/feedback regarding the use-case/implementation for this PR! Let me know if this is something you would like to discuss offline and I am open to that as well!

@alvarobartt
Copy link
Member

Hey @vrdn-23 thanks for opening this PR and apologies I'm just looking into it now! But I'll check that everything works this week, and happy to support it and add it within the next release 🤗

@alvarobartt alvarobartt self-requested a review September 15, 2025 16:30
@vrdn-23
Copy link
Contributor Author

vrdn-23 commented Sep 15, 2025

Thanks @alvarobartt for getting back to me. let me know if there are any changes I need to make!

@vrdn-23
Copy link
Contributor Author

vrdn-23 commented Oct 7, 2025

@alvarobartt just wanted to check in and see if this was still on your radar for review!

@alvarobartt
Copy link
Member

alvarobartt commented Oct 8, 2025 •
edited
Loading

Hey @vrdn-23, yes! This is something I'd like to include for Text Embeddings Inference v1.9.0, but I'd like to first make sure that some patches land, apologies for the delay 🙏🏻

Also given that we add the --served-model-name, do you think it makes sense for us to validate that the provided model parameter on OpenAI Embeddings API requests i.e., v1/embeddings, matches the actual value of either --model-id or --served-model-name unless provided as empty, similarly to how other providers with OpenAI compatible interfaces do as e.g. vLLM?

@vrdn-23
Copy link
Contributor Author

vrdn-23 commented Oct 8, 2025

Also given that we add the --served-model-name, do you think it makes sense for us to validate that the provided model parameter on OpenAI Embeddings API requests i.e., v1/embeddings, matches the actual value of either --model-id or --served-model-name unless provided as empty, similarly to how other providers with OpenAI compatible interfaces do as e.g. vLLM

I think that would be great! I think by default if the model name isn't specified in vLLM, it still serves the request, so we shouldn't have to change compatibility of the OpenAI format spec. I can add a check for the validation based on the model name specified in --served-model-name

@vrdn-23
Copy link
Contributor Author

vrdn-23 commented Oct 8, 2025 •
edited
Loading

matches the actual value of either --model-id or --served-model-name unless provided as empty,

Just to add to this, it does seem vLLM does not check for matching with --model-id if --served-model-name is specified. So maybe we check for validation like this:

  • If --model-id is specified and --served-model-name is not, validate against model-id and return model-id in response
  • If --model-id is specified and --served-model-name is specified, validate against --served-model-name and return served-model-name in response
  • If model is passed in as empty, return the --served-model-name in response else --model-id (which is what the PR currently does?)

@alvarobartt
Copy link
Member

Yes that's right @vrdn-23, I'll validate and approve ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alvarobartt alvarobartt Awaiting requested review from alvarobartt

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vrdn-23 @alvarobartt